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About the research 


Assessing the impact of research: a case study of the LSA Y Research 
Innovation and Expansion Fund 

Jo Hargreaves, National Centre for Vocational Education Research 

The purpose of this project is to apply the framework developed by the National Centre for Vocational 
Education Research (NCVER) for measuring research impact to assess the outcomes of the research 
and activities funded under the Longitudinal Surveys of Australian Youth (LSAY) Research Innovation 
and Expansion Fund (RIEF). LSAY provides a rich source of information about young people and their 
transitions from school to post-school destinations. The purpose of the Research Innovation and 
Expansion Fund was to facilitate an increase in the quantity, quality, distribution and accessibility of 
youth transitions research and analysis using LSAY data in the academic and public policy 
communities. The RIEF involved a variety of activities, including a competitive research grants 
program and a fellowship, and has led to the publication of six research reports, a national youth 
policy forum and data workshops. 

Key messages 

■ The purpose of the fund - to expand the use of LSAY data - was achieved through: 

- providing a number of researchers with the opportunity to undertake projects using LSAY data, 
many for the first time, and developing their skills to subsequently engage in other projects 
using LSAY data 

- extending the awareness of LSAY data to national and international research communities 

- imparting end-users with the knowledge and awareness of how LSAY data can be used 

- disseminating research to a wide range of policy and practitioner networks. 

■ The results of the fund have proved useful to key stakeholders. In terms of direct evidence of the 
impact on policy, one of the research reports was cited in a parliamentary legislation amendment 
bill. Research was also used to validate initiatives already underway or to inform the 
establishment of new programs. 

This study reinforces the point that the interplay between research and policy is complex. 
Nevertheless, in this case, connections between policy, practice and research have been strengthened 
as a result of the Research Innovation and Expansion Fund. This study also confirms the effectiveness 
of NCVER processes for the dissemination of research. We note the importance of a media release for 
immediate impact. Policy forums and engaging directly with policy-makers and other stakeholders can 
also play a very positive part. 


Tom Karmel 

Managing Director, NCVER 
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Assessing the impact of research 


Introduction 


The National Centre for Vocational Education Research (NCVER) is committed to ensuring that its 
research informs policy and practice in the tertiary education sector. In 2008 the organisation 
undertook a study on the impact of its research (Stanwick, Hargreaves Et Beddie 2009), which 
demonstrated that, with effort and some cost, it was possible to establish the extent to which 
NCVER research had been used. 1 A framework was developed for measuring impact against four 
domains: 

■ producing knowledge: using citations in other works and also media releases, web downloads and 
citations in parliament 

■ building capacity: assessing the extent to which the research has improved the abilities of 
researchers and stakeholders to engage with the research 

■ informing policy: judging if and how research has been useful in informing or guiding policy, 
primarily through interviews with end-users 

■ informing practice: establishing how the research has informed or guided practice, again through 
interviews with end-users. 

There has been an increased focus on research impact since our original study. The Group of Eight and 
Australian Technology Network universities, for example, have been exploring how to measure the 
impact of research, with a large trial involving 12 participating universities (as at 2012) underway. 
Understanding the different ways research can be used is fundamental to impact studies. Impact can 
be positive or negative, may or may not be recognised, can occur over short or long periods of time, 
and is highly dependent on context. In government it is shaped by the political considerations of the 
time and also public acceptability of new ideas (Levin 2003). Assessing research impact is good 
practice, however, particularly as it helps to demonstrate that government investment in research is 
providing value for money. 

The purpose of this project is to apply NCVER’s framework to a case study, whereby the impact of 
research and activities funded under the Longitudinal Surveys of Australian Youth (LSAY) Research 
Innovation and Expansion Fund (RIEF) is assessed. 


1 For the purposes of this project, research is defined as work undertaken that increases knowledge or applies 
existing knowledge in new ways. Research impact is defined as the application, use and influence of research across 
various categories. 
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About LSAY and the Research 
Innovation and Expansion Fund 

The key purpose of LSAY 2 is to increase understandings of the pathways in the lives of young people, 
particularly their transitions from compulsory schooling to further education and training and the 
labour market. The Department of Education, Employment and Workplace Relations (DEEWR) manages 
LSAY on behalf of the Australian Government and states and the territories. 

LSAY is comprised of a series of surveys focusing on the progress of young Australians as they move 
from their mid-teens to their mid-twenties. The current surveys involve large nationally 
representative samples of young people, from whom data about education and training, work, 
financial matters, health, social activities and attitudes to these and related issues are collected each 
year. The LSAY program consists of both the data collection and analysis. 

The Research Innovation and Expansion Fund (RIEF) was an initiative of the Department of Education, 
Employment and Workplace Relations intended to complement the LSAY program. NCVER was 
contracted by the department to manage and implement the LSAY analysis program. The overall 
purpose of the Research Innovation and Expansion Fund was to increase the quantity, quality, 
distribution and accessibility of independent research and analysis using LSAY data in the academic 
and public policy communities. Fund initiatives could reasonably be expected to: 

■ increase the number of researchers using LSAY data for youth transition research 

■ support the completion of high-quality and independent policy-relevant research projects using 
LSAY data, including combining LSAY with other relevant sources 

■ assist researchers to develop the capabilities needed to make full use of the longitudinal nature of 
the data and to undertake more complex or innovative forms of analysis 

■ enable low-cost access to LSAY data by researchers (especially early career researchers) and not- 
for-profit organisations 

■ strengthen connections between policy development and research - or otherwise complement the 
LSAY Analytical Program objectives. 

The key target audience for LSAY research are government ministers and senior government officers 
with responsibility for policy development and implementation in education, employment, social 
affairs, youth and disadvantaged groups. Others with a strong interest in LSAY include providers of 
services to young people (especially in the areas of career advice and transition support), social policy 
research centres, community organisations and industry. 

A particular target group of the Research Innovation and Expansion Fund was the academic research 
community, with the aim of increasing their capabilities for, and interest in, using the LSAY data for 
research. 

The Research Innovation and Expansion Fund comprised five distinct research projects, which were 
funded through a competitive grants program, and a research fellowship, as well as other activities 
such as data workshops. 


2 <http://www.lsay.edu.au>. 
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Competitive research analysis grants program 

The aim of this program was to increase the pool of researchers using LSAY data as well as to produce 
independent policy-relevant research. Research proposals were elicited by NCVER in 2008 through a 
public invitation for proposals, in a similar manner to the process undertaken for the Open Category of 
the National Vocational Education and Training Research Evaluation Program (NVETRE). The projects 
funded were chosen on the advice of a selection panel comprising representatives of the LSAY Strategic 
Advisory Committee, the Department of Education, Employment and Workplace Relations and NCVER. 
NCVER adopted specific quality assurance procedures for the management of the research projects, 
including peer review. The dissemination treatment for research commissioned through the RIEF 
program was restricted by contractual arrangements to final reports, published by NCVER and appearing 
on the LSAY website, with stakeholder notifications and media releases subject to the approval of the 
Department of Education, Employment and Workplace Relations. Five projects were funded: 

■ Education and happiness in the school-to-work transition, Mike Dockery, Curtin University of 
Technology 

■ Post-school education and labour force participation in Canada and Australia, Siobhan Austen and 
Fiona MacPhail, Curtin University 

■ Lost talent? The occupational ambitions and attainments of young Australians, Joanna Sikora and 
Lawrence J. Saha, Australian National University 

■ Against the odds: influences on the post-school success of ‘low performers’, Sue Thomson and 
Kylie Hillman, Australian Council for Educational Research 

■ From education to employment: how long does it take?, Darcy Fitzpatrick, Laurence Lester, 

Kostas Mavromaras, Sue Richardson and Lulu Sun, National Institute of Labour Studies. 

Fellowship for a new career researcher 

The primary purpose of the fellowship was to offer an early career researcher an opportunity to 
work with the LSAY data. One fellowship was funded: 

■ Returns from education: an occupational status approach, Jung-Sook Lee, University of New 
South Wales. 

Youth policy forum 

The policy forum, Young people: finding their way in a new era, was held in Melbourne on 10 
November 2009. The forum was designed to showcase how the LSAY survey data could be used in 
multiple ways to improve young people's transitions into life after schooling. 

As an excellent platform to showcase three of the five commissioned projects from the RIEF, the 
forum sought to inform policy-makers about the education outcomes for young people, the skills 
required by young people for the contemporary labour market, as well as ways to ensure that young 
people lead full and meaningful lives. 3 

Highlighting the appetite for information on youth transitions, the event attracted 289 delegates. 
Approximately 70% of those who submitted an evaluation rated their satisfaction with the event as 


3 Unfortunately a member of the National Institute of Labour Studies team was not able to attend due to ill health, 
while Siobhan Austen was not able to attend due to work commitments. 
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seven or more out of ten. The format of the event was rated by 80% of attendees as seven or more 
out of ten, and a similar proportion indicated they would attend another policy forum. Some of the 
criticisms of the event related to the lack of discussion of research results and the implications for 
policy and practice. The majority of people interviewed for this project, however, suggested that the 
research is continuing to make an impact and that the findings will be of enduring relevance: ‘when 
the forum was held some of the work presented was in developmental stages and since they have all 
been published it is just now coming to fruition’. 

The forum was an important event in terms of bringing together the researchers working on LSAY data 
and providing an opportunity for a wide range of stakeholders to become aware of and familiar with 
both the LSAY dataset and the research program. For many this was their first time engaging with the 
material and it appeared to lead to ongoing use of the research. 

Longitudinal data workshops 

The data workshops were developed and conducted by NCVER. The primary aim was to explain to 
researchers and interested policy-makers how to work with the LSAY data. Two workshops were held, 
one in Melbourne on 1 September 2009, and one in Brisbane on 2 July 2010. Further information on 
the workshops can be found in appendix A. 

LSAY data access 

To assist researchers to access LSAY data, the Research Innovation and Expansion Fund made funds 
available to cover the costs associated with accessing the data through the Australian Data Archive. 
These funds were for researchers who were not already members of the Australian Consortium for 
Social and Political Research Incorporated (ACSPRI). The initiative was advertised on both the 
Australian Data Archive and LSAY websites. 

During the period over which the RIEF ran there were only a few enquiries about data access and all 
were from researchers outside Australia. One application for funding was processed. This was for a 
researcher with the Organisation for Economic Co-operation and Development (OECD) Public 
Governance and Territorial Development Directorate, who requested access to the Y98 LSAY data. The 
Australian Data Archive Manager indicated that the other enquiries were eligible to access the LSAY 
data free of charge since they were from staff of universities that are members of the Inter-University 
Consortium for Political and Social Research. 

NCVER also maintains records of requests for access to the LSAY unit record data. Since NCVER 
commenced providing the analytical and reporting services for the LSAY program, the numbers of 
requests have steadily increased, from about 20 in 2008 up to about 40 in 2011. And while there were 
a significant number of requests following the data workshops (conducted in late 2009 and mid-2010) 
we cannot be sure if these were directly linked. 

LSAY research reference guide 

In 2004 a reference guide to the LSAY research reports, What do we know about the experiences of 
Australian youth? An easy reference guide to Longitudinal Surveys of Australian Youth research 
reports, 1996-2003 was published by the Australian Council for Educational Research. An additional 
component of the REIF was an update of this guide, such that a single, updated and current 
compendium and reference guide for LSAY research reports now exists. 


10 


Assessing the impact of research 


Methodology 

This study uses the model for assessing the impact of research, developed by NCVER in 2008, 4 
whereby impact is evaluated across the four domains of: producing knowledge, building capacity, 
informing policy and informing practice. Combinations of qualitative and quantitative impact 
indicators are used. The approach is pragmatic: using experts to provide informed opinion, and a 
broad range of citation indicators, the latter acting as proxy measures of impact. 

The model also acknowledges a number of key elements on which NCVER places high importance and 
which have been successfully implemented over many years. All of these approaches are aimed at 
maximising the impact of research. These include: 

■ establishing the priorities and purpose of the research with some involvement from end-users, 
where appropriate 

■ adopting multiple dissemination strategies, acknowledging the diverse needs of the sector 

■ engaging with stakeholders throughout the research cycle 

■ managing independent commissioned research through a competitive funding round process, 
ongoing quality assurance and advice on methodological and dissemination treatments, peer 
review, data checking and editing. 

The primary sources of information for this case study included: 

■ analysis of project information and intended impacts 

■ a bibliometric analysis, undertaken by specialist information services staff using Google, Google 
Scholar, Publish or Perish, Scopus, VOCEDplus full text and parliamentary databases. These have 
been supplemented by records of media citations held by NCVER and Google Analytics data 

■ interviews with six researchers and eight end-users (see table 1). End-users were identified either 
as a contact suggested by the researcher or by random sampling from attendees at the youth 
policy forum or data workshops. A semi-structured interview approach, designed to capture the 
various categories of impact according to NCVER’s model, was used. In addition, interviews were 
undertaken with two NCVER staff members responsible for management of the LSAY program and 
the LSAY RIEF research program. 

The detailed findings from the bibliometric analysis of each research report and the interviews 
are contained in the support document to this report available at 
<http://www.ncver.edu.au/publications/2513.html>. 


4 Assessing the impact of NCVER’s research <http://www.ncver.edu.au/publications/2152.html>. 
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Table 1 Researchers and end-users interviewed 


Researcher or organisation 


Link to LSAY RIEF program 


Mike Dockery, Siobhan Austen, Joanna Sikora, Kylie Hillman, Darcy 
Fitzpatrick, Jung-Sook Lee 

Department of Education, Employment and Workplace Relations 

Social Policy Evaluation, Analysis and Research Centre, ANU (contact 
now at University of Victoria, Canada) 

Trade Training Centre Project Manager, Polytechnic Tasmania 

Outcomes and Evaluation Division, Department of Education and Early 
Childhood Development, Victoria 


Researchers 

Suggested by researcher (Fitzpatrick) 
Attended the youth policy forum 

Attended the youth policy forum 

Attended the Melbourne data workshop and 
youth policy forum 


Polytechnic West 
CareerLink; VETNetwork 


Highlands Local Learning and Employment Network (Vic.) 


Attended the youth policy forum. 

Suggested by researcher (Dockery) ; 
contact from VETnetwork and Career 
Development Association, who invited 
Dockery to speak 

Attended the youth policy forum 


North Central Local Learning and Employment Network (Vic.) 


Attended the youth policy forum 


Caveats 

The interviews with the end-users did not involve discussion about the individual research reports but 
about interviewees’ engagement overall with the LSAY research or the forum or data workshops, 
depending on their links to the program. This is because the end-users interviewed varied in their 
knowledge and involvement with LSAY and their use of and engagement with the research projects: 
they were either more familiar with one or two specific reports or had a general awareness of all the 
research reports. Consequently, the outcomes from the end-user interviews are not reported against 
individual research reports. 

NCVER acknowledges that the sources of information and the ways of capturing them in the 
bibliometric analysis are proxies for impact; they are not ‘measures of impact’. NCVER also applies 
fairly broad definitions across the categories of impact, indicators and sources of information. Our 
bibliometric analysis involving citation indexes and web searching does not apply a strict metrics or 
ranking criterion, which is more suited to the hard sciences. Rather, for our purposes, citations 
involve references across a range of mediums, including parliamentary databases. 

Given the reference period for this case study is only one to two years from the publication date, it is 
too early for the full effects of citation to be realised. It takes time for a research report to collect 
citations, and, as our previous study revealed, the ideal reference period for a citation analysis is 
three to five years following publication. In the longer term, citations in other research documents 
will be an important source of knowledge production. 
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Findings 

Table 2 summarises the main impacts arising from the six primary LSAY RIEF reports across the four 
categories of producing knowledge, building capacity, informing policy and informing practice. 

The results in this table are then discussed under each of the four main category headings, with 
supporting data and details in the support document available at 
<http://www.ncver.edu.au/publications/2513.html>. 


Table 2 Overall summary of impacts across the six research reports 



Education 

and 

happiness 
in the school- 
to-work 
transition 

Post-school 
education and 
labour force 
participation 
in Canada 
and Australia 

Lost talent? 
The 

occupational 
ambitions and 
attainments 
of young 
Australians 

Against the 
odds: 
influences 
on the post- 
school 
success of 
‘low 

performers’ 

From 

education to 
employment: 
how long 
does it take? 

Returns from 

education: an 

occupational 

status 

approach 

(fellowship) 

Knowledge production 

Outputs One formal 

One formal 

One formal 

One formal 

One formal 

One formal 


report, 2010 

report, 2010 

report, 2011 

report, 2010 

report, 201 1 

report, 2010 

Citations 

Media release 
(x 2 NCVER 
and Curtin 
Univ.) 

Podcast 
Insight article 
45 

9 

Joint media 
release with 
Against the 
odds 

35 

Joint media 
release with 
Lost talent 

20 

Media release 
19 

11 

Capacity building 

Researcher Another team of 

Led to a 

Led to 

The research 

Opportunity for 

Having worked 

perspective 

researchers 

successful 

subsequent 

team has 

the research 

on 


from NATS EM 

International 

work and 

extended the 

team to engage 

predominantly 


built on this 

Research 

proposed future 

work by running 

with the LSAY 

qualitative 


research; 

Linkage Grant 

work by the 

the same 

dataset for the 

research the 


researcher 

through the 

research team, 

models on high 

first time, and is 

fellowship gave 


gained further 

Canadian 

both in terms of 

performers 

now one of the 

the researcher 


funding; 

Government, 

research grant 

(PISA 

datasets they 

the opportunity 


researcher had 

extending the 

applications and 

forthcoming); 

will consider 

to engage with 


discussion 

work to 

collaborations; 

had the 

using to answer 

the LSAY data 


about doing 

consider how 

the OECD 

opportunity to 

research 

and gain 


future papers 

post-school 

applied for 

talk to other 

questions in the 

knowledge 


on this work 

education 

access to the 

countries 

future 

about 

End-user 

with visiting choices LSAY data interested in the quantitative 

researchers translate into following a benefits of research from 

from overseas labour market presentation by building on NCVER 

participation the researcher longitudinal researchers, 

on the value of surveys which has led to 

using the data subsequent 

involvement in 
quantitative 
research 
projects 

Greater awareness of LSAY data and research and how it could be applied in their own organisations. 

perspective 

Supported professional development activities. 

Developed confidence in their knowledge about LSAY research more broadly, and capacity to share this 


information amongst networks. 
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Table 2 Overall summary of impacts across the six research reports (continued) 5 


Education 

and 

happiness 
in the school- 
to-work 
transition 

Post-school 
education and 
labour force 
participation 
in Canada 
and Australia 

Lost talent? 
The 

occupational 
ambitions and 
attainments 
of young 
Australians 

Against the 
odds: 
influences 
on the post- 
school 
success of 
‘low 

performers’ 

From 

education to 
employment: 
how long 
does it take? 

Returns from 

education: an 

occupational 

status 

approach 

(fellowship) 

Contribution to policy 

Researcher Not aware of 

Not aware/ 

Not aware/ 

Not aware/ 

Cited in the 

Not aware/ 

perspective/ direct impacts, 

none cited 

none cited 

none cited 

Social Security 

none cited 

citation but believes it 

analysis has been used 

in forums within 
the VET and 
career 

development 
fields, and in 
DEEWR and 
Productivity 
Commission 
reports 




Legislation 
Amendment Bill 
and media 
release 
(Senator Chris 
Evans 2011 
‘Research 
confirms a good 
education is the 
key to getting a 
job’) 



End-user The research has informed policies on rural and remote education and the governance of the Trade 
perspective Training Centres in Tasmania (Polytechnic Tasmania) and in a policy framework around youth transitions 
(Department of Education and Early Childhood Development). 

DEEWR used the research in a paper for APEC and in a cabinet document looking at disadvantaged youth; 
in both cases the findings from the research were used to set the scene about youth transitions. 

Contribution to practice 

Researcher While the researchers were unable to cite specific examples, they have all presented at conferences or 
perspective practitioner networks, sometimes at the request of those networks. 

End-user Informed the way the Trade Training Centres at Polytechnic Tasmania established partnerships between 
perspective training providers and regional schools. The research also had practical implications for supporting students 
and for how they set up vocational education in schools. 

Informed the Youth Partnerships Initiative, of which the North Central Local Learning and Employment 
Network is one of eight trial sites. 

Knowledge production 

The most common forms of output from the research were formal reports and presentations (table 2), 
consistent with the outputs proposed as part of the dissemination strategies. Table 3 gives more 
detailed results of the bibliometric analysis. Education and happiness in the school-to-work transition 
had the most varied outputs from the research and also had the greatest number of citations. 

Captured in the ‘Other’ citations data (table 3) are a number of presentations the researchers gave. 
Some of these presentations occurred as a direct result of requests from organisations working in the 
field of youth transitions, such as the Career Development Association of Australia. There has also 
been extremely strong international interest in all of the research, with presentations in Canada, 
Ireland, Sweden, France, Portugal, Germany and Hong Kong. 

It is particularly interesting to note that From education to employment: how long does it take? was 
cited in the Social Security Legislation Amendment Bill in November 201 1 and by Senator Chris Evans, 
the Minister for Tertiary Education, Skills, Jobs and Workplace in a parliamentary media release. 


5 Note that details are expanded further in the sections below and also in the support document. 
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Table 3 Summary of citations across the RIEF research reports 6 



Education 

Post-school 

Lost talent? 

Against the 

From 

Returns from 


and 

education 

The 

odds: 

education to 

education: an 


happiness 

and labour 

occupational 

influences 

employment: 

occupational 


in the school- 

force 

ambitions and 

on the post- 

how long 

status 


to-work 

participation 

attainments 

school 

does it take? 

approach 


transition 

in Canada 

of young 

success of 


(fellowship) 



and Australia 

Australians 

‘low 

performers’ 



Publication 

19 April 2010 

21 September 

12 January 

21 September 

29 June 2011 

16 December 

date 


2010 

2011 

2010 


2010 

Where cited (as at 31 March 2012) 





Journal article 

- 

1 

1 

1 

- 

- 

Research 

reports 1 

4 

3 

4 

4 

3 

2 

Book chapters 

- 

- 

- 

1 

- 

- 

Conference 

papers 

3 


1 

1 

- 

- 

Parliamentary 

databases 

- 

- 

- 

" 

2 

- 

Other 2 

6 

3 

7 

4 

- 

3 

Media 

32 

2 

19 

9 

14 

6 

Total 

45 

9 

32 

20 

19 

11 

Google Analytics website data (from report publication date to 31 March 2012) 



Total unique 
page views 3 

2202 

1138 

2619 

2727 

2189 

781 

Media release 

386 

- 

752 

752 

361 

- 

Google Analytics VOCEDplus data (from report publication date to 31 March 2012) 


Full record 
display unique 
page views 

10 

12 

10 

7 

8 

9 


Notes: (1) Research reports include citations in NCVER publications. 


(2) ‘Other’ includes citations in a variety of outlets such as At a glances, research degree theses, edited volumes, resource 
materials and other presentations. 

(3) Unique page view: when a visitor to the website views the same page more than once during the time they spend on 
the website, Google Analytics aggregates this number so that it is recorded as the one unique view of that page by that 
single user. 


Media 

While media coverage is not a precise indicator of impact, it does provide a general indication of a 
topic resonating across a broader audience. We need to be mindful that there are a number of 
contributing factors that determine what may or may not ‘grab’ attention at any particular time, and 
which can also be affected by other events. Nevertheless, a media release does generate substantial 
additional immediate interest in a report, as the Google Analytics data below demonstrate (table 4). 

NCVER maintains its own records of media coverage. This primarily relates to articles in national and 
state/territory newspapers, radio and articles in national training or industry-related magazines or 
newsletters. Most coverage tends to occur on publication release. Education and happiness in the 
school-to-work transition generated the most media coverage, followed by Lost talent? The 
occupational ambitions and attainments of young Australians. 

The last impact study was undertaken by NCVER in 2008 and it is interesting to note that since that 
time the level of citations across all media outlets has substantially increased. This may suggest that 


6 The detailed citations for each of the LSAY RIEF research reports are provided in the accompanying support document. 
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general media interest in NCVER research is rising. However, the other contributing factor in this 
study is that the media routinely seizes information on youth issues as a matter of course because this 
area generates a great deal of interest with the general public. 


Google Analytics 

Since NCVER’s last impact study the organisation has made improvements to the collection of website 
data using Google Analytics. Google Analytics is a free web analytics tool that NCVER has used to 
collect and analyse usage statistics on the NCVER and LSAY websites. It tracks how visitors interact 
with the websites, including where they come from, how many visit the websites, who they are (user 
groups), how long they spend on the website, what pages they visit and what publications they view 
or download. Users are able to access LSAY reports via the LSAY and NCVER websites; and data are 
provided for both websites. The date of the reporting period of this study extends from publication of 
the research report to 31 March 2012. The detailed data appear in the support document to this report. 

Of the groups we can identify, government users, research organisations and ‘other’ are the most 
frequent visitors 7 to the LSAY RIEF research reports on the websites - in varying order across each of 
the reports. The terms most used to clarify the ‘other’ category are: 

■ Most used, in order: 

- student (+ teacher, TAFE, university) 

- university (+ student) 

- NGO 

- consultant (+ VET consultant) 

- community (+ sector, organisation) 

- organisation (+ community, international, non-government organisation). 

■ Next most used, in no order: 

- apprenticeship centre 

- group training 

- not for profit 

- personal 

- private 

- professional (association, organisation, services) 

- RTO (registered training organisation) 

- software developer 

- private (company, research, study). 

The Google Analytics data reveal that over half of those looking at the publication summary page go 
on to view the full report. 

Most of the users accessing the publications have come from Australia, but the data show users from a 
wide range of countries, with From education to employment: how long does it take? generating 


7 Apart from the category ‘guest’, which can be defined as a user who has come to the website for the very first time, 
or whose previous visit(s) have not been remembered by the website, or they are yet to log in. 
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interest across 31 countries. Not surprisingly, 7% of users accessing Post-school education and labour 
force participation in Canada and Australia were from Canada. 


Most user activity occurs within the first one to two weeks of a report being published, but analysis of 
Google Analytics data over time can reveal: 

■ spikes or renewed interest in a report, perhaps as a result of their relevance to specific external 
events or due to information about them being posted on other websites such as Australian Policy 
Online or in NCVER News. This was the case with Post-school education and labour force 
participation in Canada and Australia and Lost talent? The occupational ambitions and 
attainments of young Australians 

■ whether a report receives regular and relatively consistent viewing, as has been the case with 
Education and happiness in the school-to-work transition during the reporting period. 

The data confirm the importance of media releases and email alerts to generate interest, bringing the 
research to the attention of policy-makers and practitioners. Based on total number of page views, 
the top four most popular RIEF reports (table 4) all had an associated media release when published, 
whereas the other reports did not (as agreed between the Department of Education, Employment and 
Workplace Relations and NCVER, based on dissemination plans and contractual requirements). 


Table 4 Google Analytics top four RIEF publications 


RIEF publications 

Release date 

Total # of 
page views 

Media release 

Against the odds: influences on the post-school success 
of ‘low performers' 

21 September 2010 

2727 

Referenced in 
Lost talent? 
media release 

Lost talent? The occupational ambitions and attainments 
of young Australians 

12 January 201 1 

2619 

Yes 

Education and happiness in the school-to-work-transition 

19 April 2010 

2202 

Yes 

From education to employment: how long does it take? 

29 June 2011 

2189 

Yes 


General observations about dissemination 

A wide range of dissemination strategies, both print and other, are an important component of the 
applied research process adopted at NCVER. As part of the interviews, respondents were asked to rate 
how important particular forms of dissemination were to facilitating impact, on a five-point scale 
ranging from one (of no importance) to five (of great importance). 

While some respondents qualified their answers, the average scores are fairly evenly distributed 
across the various avenues of dissemination (table 5). In the previous impact study, media coverage 
and academic publications stood out for not being rated as highly. 


Table 5 Importance to impact of various types of dissemination 


Type of dissemination 

Score 

Research reports 

3.94 

Seminars and conferences 

3.88 

Briefing papers 

3.91 

Media coverage 

3.44 

Academic publications 

3.56 

Discussions with policy-makers 

3.69 


NCVER 


17 


In a comparison of ratings with the end-users, most researchers in this study not surprisingly tended 
to give a higher value to the impact of academic publications. A few respondents concurred that 
media coverage is good for prompting people to take notice but that it doesn’t actually mean they 
use the findings. 

When asked what dissemination strategies end-users find particularly useful in raising awareness and 
leading to use of the research findings - or data sources such as LSAY - the following were the most 
commonly suggested: 

■ presentations and events 

■ short (two-page) summaries, or briefing papers 

■ email alerts, preferably with a brief summary about the research and which link directly to 
the publication. 

The overall consensus from the interviews was that dissemination strategies that make the research 
easily accessible, in plain English and focusing on practical outcomes were the most favoured. A key 
challenge is ensuring that briefing papers are pitched to the right audience. Another critical element 
is whether the research answers the question that policy people are trying to answer at that time — 
regardless of good dissemination the topic needs to be relevant and timely. 

Capacity building 

In all cases the researchers were able to cite specific examples of capacity building, in terms of 
extending their research through further research grants and/or collaborations, as well as engaging 
with the LSAY dataset. 

Three of the five research projects involved researchers who were new to working with LSAY data. In 
addition, the fellowship specifically targeted a researcher new to LSAY. This researcher experienced a 
significant amount of capacity building, since he had moved from a purely qualitative research 
background to engagement in quantitative methodologies. Some of these researchers commented that 
they found using the dataset challenging because of its complexity, but overall a worthwhile exercise. 

One of the projects led to a successful application for an International Research Linkage Grant 
through the Canadian Government. The grant was awarded to enable them to extend their work on 
the post-school education choices of young women in Canada and Australia and to consider how those 
choices translated into the labour market. Another principal researcher used their research project to 
demonstrate the value of using LSAY data to the Directorate for Education at the OECD, and as a 
consequence the OECD applied for access to the data. An unexpected impact for one research team 
was that they were given the opportunity to begin discussions with other countries that want to 
explore research questions that can be addressed by using a longitudinal dataset such as LSAY. 

An accumulated body of work may generate greater impact, but, as one researcher noted, the 
capacity building aspects of single projects can be significant. They generate ‘further opportunities 
for the researchers’ and directly assist in building a body of work around specific topics. Moreover, 
some of the researchers were involved in the consultations to establish a new set of research 
priorities for the LSAY program, which will lead to more targeted research in the future and build on 
the work already done. 

There was also clear evidence of capacity building amongst the end-users as a direct result of 
initiatives within the RIEF, particularly through access to the research reports, having the opportunity 
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to explore the data in a workshop, or attending the youth policy forum. Many became much more 
aware of LSAY data and associated research after the forum. Their attendance at the forum supported 
their professional development activities, while encouraging them to increase and share their 
knowledge about LSAY research more broadly. Although some were already using the research before 
attending the youth policy forum, the forum made them more aware of the relevance of the research 
and increased their confidence in using it. Many end-users play a key role in increasing research 
impact, by filtering and helping to tailor findings to increase their relevance and accessibility to a 
particular audience. One end-user commented: 

The forum gave us exposure to a range of approaches to conceptualising research into youth 
transitions, and some interdisciplinary perspectives. The forum also led to forming a range of 
connections with people who work with the LSAY data. (University of Victoria, Canada, end-user) 

Following the forum another end-user offered support to an early career researcher in her 
organisation in applying for an NCVER Community of Practice research scholarship to conduct specific 
research on youth transitions for their region. 8 This project conferred ongoing development benefits 
in terms of the acquisition of research skills in their organisation. 

Contributing to policy 

Many of the researchers were unable to confirm whether the research had informed policy. One of the 
challenges this raises is how to encourage and equip researchers to become more engaged in tracking 
the influence of their work beyond academia and to report on the outcomes and ongoing influence of 
their research in policy and practice. 

As noted in the section relating to citations, this case study did find evidence of impact on policy. 
From education to employment: how long does it take? was cited in Social Security Legislation 
Amendment Bill (Second Reading of House of Representatives, Hansard, 23 November 2011) and in the 
associated media release: ‘Research confirms a good education is the key to getting a job’ (2011). 
Citations in journals are often used in academia as an easy and popular measure of impact, but when 
looking for research’s influence on policy we find that citations in parliamentary legislation such as 
Hansard have significance. 

Several of the end-users were able to provide examples of where the research has been used in 
policy. It has informed policies on rural and remote education and on the governance of the Trade 
Training Centres in Tasmania (Polytechnic Tasmania) and in a policy framework relating to youth 
transitions (Department of Education and Early Childhood Development). 

The Department of Education, Employment and Workplace Relations used the research in From 
education to employment: how long does it take? in a paper for APEC and in a Cabinet document 
addressing disadvantaged youth. In both cases the findings from the research were used to set the 
scene on youth transitions. They noted that this kind of research has an educational function, with 
the Department of Education, Employment and Workplace Relations commenting about the Fitzpatrick 
et al. report that it ‘help[ed] to keep people who make policy decisions better informed, but given 
the type of research question it may be unlikely to lead directly to policy change’. 


Hegol, L 2012, Transition to post-Year 10 education in regional Tasmania, NCVER, Adelaide. 
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All people interviewed for this project recognised the difficulties inherent in tracking the ways by 
which research directly or specifically influences policy. One of the NCVER staff members with 
oversight of the LSAY program noted that the primary stakeholder groups responsible for youth affairs 
are not a cohesive group. This presents additional challenges in terms of reaching the various target 
audiences. On the other hand, research findings relating to youth issues resonate well with the media, 
which gave good coverage to some of the research reports. 

This study was undertaken only 12 months after the final research report from the LSAY RIEF suite was 
published. It is possible its impact on policy is yet to eventuate. 

Regardless, several end-users commented on the key findings from the research as being useful - in 
either providing new information and concepts or strengthening and validating the case for existing 
policies. They also noted that the forum was accessible to people from a policy background. One end- 
user stated: 

The more informed you are about the issues then the better you can influence policy, so rather 
than being able to provide specific examples it is more about the underpinning knowledge gained 
from the research that you take to the policy debate. 

(Highlands Local Learning and Employment Network) 


Contributing to practice 

A couple of the end-users were able to cite examples of where the research has influenced or 
informed practice. 

In one case it informed the way the Trade Training Centres at Polytechnic Tasmania established 
partnerships between training providers and regional schools. The research also had practical 
implications for supporting students and for how vocational education in schools could be set up. 

In another case, an end-user believed that the research has informed the Youth Partnerships 
Initiative, of which the North Central Local Learning and Employment Network is one of eight trial 
sites. The Youth Partnerships Initiative is a Victorian Government program aimed at designing and 
testing new ways for the education, youth and family support, justice, homelessness and mental 
health sectors to work more collaboratively to support individual young people experiencing 
problems. In particular the LSAY research was used to refine the definition of vulnerability and to 
inform specific project development. This end-user also advised they are aware generally of the LSAY 
research being referenced in numerous publications, for example, those emerging from the Youth 
Affairs Council of Victoria. 

Following his presentation at the youth policy forum, Mike Dockery was asked to present at several 
practitioner networks, including VETnetwork and the Career Development Association of Australia. 

The VETnetwork professional development event, ‘School to work transitions’ held in Perth in August 
2011, attracted 40 participants from TAFE institutes, government schools, education departments and 
group training organisations across metropolitan and regional areas. It is an example of the ‘spin-offs’ 
generated from a presentation at one event and how research findings can be extended to people ‘on 
the ground’. 

Some of the end-users commented that, rather than directly influencing a change in practice, the 
research may be used primarily for information. In part, this is a reflection on the timeliness of the 
research, with one end-user commenting that, ‘while the research is interesting we are already aware 
of the issues and taking measures to address it’. 
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Many continue to engage with the research and share that knowledge by drawing other people’s 
attention to the research findings, either via informal processes or specific references in papers and 
policy- related documents. Several were able to cite specific examples of how they have subsequently 
used the knowledge gained from the youth policy forum and the research, for example, to promote 
the research findings to the principals and deputy principals of the schools they work with. 
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Conclusion 


The overall purpose of the program of activities associated with the Research Innovation and 
Expansion Fund was to increase the quantity, quality, distribution and accessibility of independent 
research and analysis using the LSAY data, particularly in the academic and public policy communities. 

This study has shown that the RIEF has provided the opportunity to create a more self-sustaining 
critical mass of researchers with a relationship to LSAY, with the majority of researchers funded in 
the analysis grants program being new to research making use of LSAY data. Six high-quality 
independent research reports that made use of LSAY data were published as a direct result. 

The published reports subsequently led to researchers from other organisations, nationally and 
internationally, exploring the ways by which LSAY data could be used. The citation analysis and 
interviews with researchers revealed several examples of other research teams or the researchers 
themselves continuing their work on LSAY data. Some of these research projects combine LSAY with 
other relevant sources such as the Household, Income and Labour Dynamics in Australia (HILDA) 

Survey and NCVER data. 

The youth policy forum and data workshops reached a wide range of academic, public policy and 
practitioner communities. The end-users interviewed for this project varied in their knowledge about 
LSAY and the research reports. One had been involved with LSAY for over ten years, while others were 
exposed to LSAY and the research for the first time through their attendance at the forum. All 
commented that the forum was a great opportunity to engage directly with the researchers, explore 
additional questions and other work, as well as establish networks and collaborations. 

The end-users of the research in this study identified examples of how the research findings have 
been promoted to broader networks and how the LSAY data have been used for youth transition 
research in their own practice. 

The Research Innovation and Expansion Fund deliberately adopted a research model similar to that 
of the National Vocational Education and Training Research and Evaluation Program; that is, the 
external commissioning of research . 9 This was seen as a positive model, with one person from the 
Department of Education, Employment and Workplace Relations expressing the view that, given that 
the RIEF resulted in a broader range of academic researchers engaging with LSAY, this model could 
be replicated. Independent evaluations of the LSAY program conducted for the department also 
highlighted the success of the RIEF in enabling a wider range of researchers to access and use the 
LSAY data. 

The data workshops and the concomitant ability to work closely with NCVER staff enabled researchers 
to develop capabilities in terms of both research with longitudinal data and the adoption of more 
complex or innovative forms of analysis. The fellowship recipient, who moved from a purely 
qualitative research background to quantitative methodologies, is a case in point. 

Several researchers and end-users commented that the LSAY data require a substantial amount of 
promotional work, as the dataset is not well known. Some were surprised that it is not used more 
often. Another stated that social science students are not oriented towards the use of survey data as 


9 <http: / /www. ncver.edu. au/research/about.html#National_VET_Research_and_Evaluation_(NVETRE)_program>. 
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part of their research training, so there is a need to specifically promote the use of LSAY as a good 
source of information (for example, to researchers beyond the economics or social policy disciplines, 
such as to the Australian Psychological Society and other professional bodies). One end-user 
mentioned the Economic and Social Outlook conferences run by the Melbourne Institute, which 
showcase the HILDA data, suggesting this might be one way to consider bringing the LSAY data to the 
notice of more people. 

Another suggested identifying research institutes in universities and government departments aligned 
to topics to which LSAY might be relevant and contacting them directly. Another end-user specifically 
commented that the workshops, which allowed people to work through the data and showed them 
how it could be used, offered the best solution to its promotion. 

One researcher remarked that a great deal of work on education and young people ‘falls between the 
cracks’. Her suggestion was to promote the messages to teachers in schools rather than to a policy 
audience: ‘focus on tailoring the outcomes of the research to things teachers can actually do and have 
control over, which will not be the same thing as policy-makers’. 

Another researcher highlighted the importance, to ensure maximum impact from the research, of 
getting policy people engaged in the research. It is not only NCVER’s role to promote research to the 
government but also for the researchers themselves to maintain ties with policy circles. 

This study reinforces the point that the interplay between research and policy is not straightforward. 
Nevertheless, connections between policy, practice and research do occur, and in this case have been 
strengthened as a result of the Research Innovation and Expansion Fund, with researchers being 
invited to speak at practitioner events, citations in a parliamentary legislation and end-user examples 
of engagement with and use of the research. 

In addition, the research has contributed more broadly to the three higher-level research priorities of 
the LSAY program. These are aimed at: understanding the factors that impact on participation and 
completion in education and training; investigating whether the skills from, and pathways through, 
education and training are well matched with labour market demands; and gaining better insights into 
the wellbeing and social activities of young people. 

The primary purpose of the Research Innovation and Expansion Fund was to increase the capacity of 
researchers and end-users working with LSAY data, and this study has shown clear examples of where 
this has occurred. As a model, the RIEF combined a number of initiatives, which collectively had 
impact in terms of raising the profile of LSAY and increasing the quantity and quality of research. 

This study has also confirmed that NCVER is on the right track with its dissemination of research. The 
capacity of researchers to purposefully engage in the end use of their work was highlighted, and 
NCVER will continue to promote this objective. NCVER will also continue to periodically select themes 
of research for impact studies, thereby continuing to build more robust examples of where research is 
influencing policy and practice. 
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Appendix A: data workshops 

A number of data workshops were held during 2008-10: 

■ In late February 2009, NCVER staff facilitated a data workshop for members of the National 
Institute of Labour Studies research team with the aim of helping them interpret the LSAY data for 
their research project. 

■ In early September 2009, four NCVER staff, together with Pam Millwood from Wallis Consulting 
Group, facilitated a workshop for researchers on the LSAY data. The purpose of the workshop was 
to broaden the use of LSAY data and to increase awareness of the LSAY program more broadly as a 
source of empirical evidence for informed policy-making. The workshop also provided an 
opportunity for NCVER to showcase the new LSAY cohort report structure and format, as well as 
the revised LSAY technical documentation. There was a high level of interest in the workshop, 
with the maximum number of 30 places filled. Workshop participants were current and potential 
users of the LSAY data from the academic community, government departments, non-government 
organisations and other research organisations. The event was very successful, with positive 
feedback on the day from participants. 

■ In response to a request from the National Senior Officials Committee (NSOC) representative on 
the LSAY Strategic Advisory Committee (SAC) and the Executive Director, VET Policy Development, 
Queensland Department of Education and Training, at the November 2009 SAC meeting, two LSAY 
data workshops were held in Brisbane on 2 July 2010. The first, 'Getting to know LSAY', was aimed 
at individuals with an interest in youth transitions who wanted to learn more about what LSAY has 
to offer. The other workshop, The Ins and Outs of LSAY Data’, was directed at researchers with a 
quantitative background who were interested in youth transitions and in using longitudinal data to 
investigate this subject matter. The responses however were mixed, with some of the attendees 
attending the wrong workshop, based on their skills and backgrounds. 

■ In response to another request from a representative on the LSAY Strategic Advisory Committee, in 
July 2010 NCVER hosted a seminar at the Department of Education and Training, NSW on how to 
access and use LSAY statistics and resources. 
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Support document details 

Additional information relating to this research is available in Assessing the impact of research: a case 
study of the LSAY Research Innovation and Expansion Fund - support document. It can be accessed 
from NCVER’s website <http://www.ncver.edu.au/publications/2513.html>. It contains detailed 
findings from the bibliometric analysis of each research report and the interviews. 


26 


Assessing the impact of research 



Longitudinal 
Surveys of 

Australian Youth 


National Centre for Vocational Education Research Ltd 

Level 1 1 , 33 King William Street, Adelaide, South Australia 


*NCVER 

PO Box 8288, Station Arcade, SA 5000 Australia 



Telephone +618 8230 8400 Facsimile +6 1 8 82 1 2 3436 


Australian Government 

Websitewww.ncveredu.au Email ncver@ncveredu.au 


Department of Education, Employment 
and Workplace Relations 



